REMARKS 



The claims have been amended in order to cancel claims 1-13, to add claims 14-26, to 
eliminate multiple dependent claims and claims improperly depending from multiple 
dependent claims and to otherwise conform the claims to U.S. practice. Care has been taken 
to ensure that no new matter is added to the text. 

Paragraph 6 has been amended to remove references to the claims in the specification. 
Applicants are submitting herewith a substituted Specification. Care has been taken to ensure 
that no new matter has been added. 

Entry and favorable consideration prior to consideration are respectfully 

requested. 
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THREE-DIMENSIONAL RECONSTRUCTION OF SURFACE PROFILES 



Cross Reference to Related Application 

[0001] This application is a national stage of PCT/EP2004/Q 10729 filed September 24, 
2004 and based upon DE 103 46 481.6 filed October 2. 2003 under the International Convention. 

BACKGROUND OF THE INVENTION 

Field of the invention 

[0002] The invention concerns processes for detecting and reconstructing structures of 
surfaces, such as for example elevations, grades, or dips. 



[0003] Various processes are known, in which profiles of surfaces can be detected. Two 
different principles can be distinguished: on the one hand, those processes in which the surface 
to be characterized is directly sensed, for example mechanically, and on the other hand, the 
contactless techniques. 

[0004] Mechanical sensing instruments, so called "profilometers," extend a sensing probe or 
needle in the manner of a grid over the surface of the sample and detect thereby the vertically 
changing surface profile via the changing positions of the needle tips. They are frequently 
employed where the objects to be distinguished and the there upon to be detected surface change 
area is not very large, since on the one hand a precise positioning of the sample piece under the 
sensor is necessary, and on the other hand the scanning type sensing of large areas is associated 
with a correspondingly large expenditure of time. 

[0005] Contactless sensing processes use for example the reflection of ultrasound (the 
principle of the echolocation) or are based upon optical (for example laser scanner) or radar 
technical processes. Depending upon the area of employment, one or the other process may be 
more advantageous. Ultrasound processes are not suited for all environments nor for very large 
distances between object and sensor. For very large objects (for example earth surface) very 
precise elevation information is provided using for example radar altimetry; this however 
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requires a technically sophisticated sensor and requires precise determination of sensor positions. 
Optical processes, in particular laser-based, require in general a relatively high calibration 
expense. A special optical process is stereometry, in which the surface to be examined is 
recorded using various views or perspectives and, from the evaluation of the slight stereoscopic 
deviations, the structures of the surface can be determined. 



[0006] The present invention begins with the known optical process for detecting and 
reconstructing surface structures. It is concerned with the task of providing a new process, 
which is characterized by having broad applications. This task is solv e d by tho proc e ss 
according to Claim 1. — Further details and advantag e s of th e invention arc sot forth in the 
d epe ndent claims. 



[0007] The inventive process is set forth in greater detail in the following on the basis of 
illustrative embodiments. Therein reference is made to the figures and the therein shown 
reference numbers. There is shown: 



SUMMARY OF THE INVENTION 



BRIEF DESCRIPTION OF THE DRAWINGS 



Fig. 1 



image of a part of the moon surface, 



Fig. 2 



detail view from Fig. 1 with varying light relationships 2.1 and 2.1, 



Fig. 3 



extraction of the shadow fields from Fig. 2.1 and 2.2, 



Fig. 4 



accentuating the boundary lines of the shadow fields in Fig. 2.1 and 2.2, 



Fig. 5 



supplemental accentuation of the course or contour of ridges and boundary lines 
about the flanks of the elevations, 



Fig. 6 



profile reconstruction of the examined regions, 
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Fig. 7 image of an approximately 0.5 cm high ridge on a metallic surface; light 

impinging from left and right, 

Fig. 8 calculated high profile of the ridge from Fig. 7, 

Fig. 9 perspective representation of the reconstructed grade, 

Fig. 10 image of a work piece surface with small defect; light impinging from left and 
right, 

Fig. 1 1 calculated height profile of structures on the work piece surface of Fig. 10, 

Fig. 12 image superimposition of color selective images of a surface contour, 

Fig. 13 stamping on a metal work piece; images with various light incident angles, 

Fig. 14 section from Fig. 13 with light incident directions (arrows) and reconstructed 

images, and 

Fig. 15 reconstruction of the stamping according to the inventive process with integrated 

shape-from-shading. 

DETAILED DESCRIPTION OF THE INVENTION 

[0008] The inventive process evaluates various images of the same area of the surface to be 
examined, wherein the images are recorded from a nearly vertical perspective with strongly 
differing light incident angles from different sides. The invention takes advantage of the fact 
that, in particular in the case of grazing or glancing light incident angle, that is, with an angle to 
the surface of approximately 10° and less, elevations or recesses on the surface can cast a 
significant shadow. The shape of the cast shadow is therein obviously directly dependent upon 
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the form of the illuminated profile, that is, width, height and depth of the structure on the surface 
correlated with various contours of the cast shadow. Besides the analysis of the shadow 
contours, the inventive process also takes into consideration the various light reflections of 
appropriate areas of the surface, for example, for determining the course or contour of a ridge 
surface. 



[0009] In the simplest case one records two images using grazing light, whereby the light is 
shined upon the structure from various, (for example opposite) sides with a flat angle. 
Preferably, however not necessarily, the camera position remains unchanged relative to the 
surface, since in this case the evaluation is simplified (identical pixel-coordinates for identical 
structures on the surface). If however the camera position changes, then these positional changes 
can be taken into consideration by appropriate corrections during the evaluation. 

[0010] The thus obtained images were super-positioned in a first step for the further 
evaluation, such that corresponding image points (pixels) were superimposed (imaged structures 
covered each other). In the next step pixels corresponding to gray values were calculated into 
new image points by quotient formation. From two starting images with light intensity values Ii 
or as the case may be I 2 there resulted thus two "quotient images" with pixel values Ii/I 2 or, as 
the case may be, h/I\. By the quotient imaging areas with small albedo (reflection) (on other 
otherwise flat areas of the surface), which in the case of evaluation of only one starting image 
could be result in the light intensity differences being erroneously evaluated as a shadow, is 
annulled, since both starting images (the same angle of incidence of the light source is a 
precondition) for this area exhibit the same light intensity value, which is rectified during 
quotient formation. The quotient images then obtain - strongly contrasted - the cast shadow 
areas separated according to light incidence, that is each quotient image I1/I2 or as the case may 
be I 2 /Ii provides the areal of the cast shadow, which occurs respectively upon illumination of 
only one side. 



[0011] In the next step the contours of the cast shadow-areals are extracted. This occurs for 
example by a "binary connected component" (BCC) - analysis of the quotient images (see E. 
Mandler, M. Oberlaender: One Pass Encoding of Connected Components in Multi-Valued 
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Images, IEEE Int. Conf. on Pattern Recognition, pp. 64-69, Atlantic City, 1990). The thus 
obtained contours can be refined by a flattening processes, for example B-Spline-Interpolation 
(See D.F. Rogers: An Introduction to NURBS - With Historical Perspective, Academic Press, 
San Diego, 2001), in order to further increase the precision of the profile reconstruction beyond 
the pure resolution of the images (Image Point - Grid Pattern). Alternatively, the contours 
determined using for example s BCC- Algorithm could be used as initialization for a figmentation 
of the shadow boundaries on the starting images with active-contours (See D.J. Williams, M. 
Shah: A Fast Algorithm for Active Contours and Curvature Estimation, Computer Vision, 
Graphics Image Processing, 55, pp. 14-26, 1992). 



[0012] If the contours of the cast shadows are separately determined for each light incident 
angle, then in the next step the course or profile of a rise of an elevation, or even steps, on the 
surface can be determined. For this it is presumed, that for example the side incident of light 
with a shallow angle from right forms a leftward cast shadow of a rise in the image of a shadow 
boundary, of which the right edge is formed by the rise of the elevation itself. The same applies 
analogously for the second image (illumination from left). The comparison of corresponding 
edge areas of shadow contours of both images provides for a grid pattern the same contour 
segments in both images (right edge of a shadow contour in Image A corresponds to the left edge 
of corresponding contour in Image B). A step on the surface provides, in distinction, a shadow 
contour on only one of the images. 

[0013] For each point of a rise determined in this manner (or a step) the associated shadow 
length (stretching of a cast shadow parallel to the incident angle of the light source) can be taken 
directly from the extracted shadow contour. With known incident angle there is produced 
therefrom the elevation of the rise point (or, as the case may be, edge of a step) and therewith the 
height of the grid (or as the case may be the step). 

[0014] In the last step the tilt surfaces on the surface are determined. The surface areas 
tilted towards the light source are significantly lighter in the images than the planar surface areas. 
These regions can be extracted from the image using, for example, a BCC-algorithm. For this, as 
selection criteria for the starting images, first a respective light intensity value Qi or as the case 
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may be 0 2 is determined as threshold value, and then therewith only the regions are selected of 
which the image points are lighter than the predetermined threshold value 0i or as the case may 
be 02 • Supplementally it is taken into consideration which of the thus selected areas are 
illuminated respectively only one of the two starting images and not in the other, that is, for the 
respective quotient image there applies Ij/I 2 > 0o (or as the case may be I 2 /Ii > 0o ), wherein 0 O is 
a predetermined threshold of the quotient image. The areas identified in this manner as tilt 
surfaces in this manner can be extracted again by means of BCC-algorithm and/or active contour 
process as contour gradients. In connection with the determined elevation profile (rise) the angle 
of tilt can be estimated (for example a flank of a rise or grade) and therewith finally the profile of 
the elevation can be reconstructed. 



[0015] The contour of the flank is determinable herein only relatively approximately (linear 
increase). Changes of the course of the slope or also flat structures (low depressions, slight rises 
with small tilt) can however be determined by "shape-from-shading" (see X. Jiang, H. Bunke, 
Three Dimensional Computer Vision, Springer Publishers, Berlin 1997). Therein slight changes 
of the reflected light intensity are evaluated, in order, beginning with known geometric 
arrangement between camera, object and light source, to arrive at conclusions regarding the 
respective tilt of the reflected area. If one combines the inventive process with the shape-from- 
shading method, then surface profiles can be reconstructed with high quality. One example 
thereof is shown in Fig. 13-15 (explanation can be found under Illustrative Example 3). 

[0016] In particularly preferred manner, first the surface profile of the surface to be 
reconstructed is determined with suitable initialization by means of the shape-from-shading 
method. Subsequently, then, in particular advantageous manner, the angle between each flat 
element and the light incident direction applicable for the shadow image is so multiplied with a 
constant factor, that the average height or elevation differential in the reconstructed profile 
corresponds to the average height differential obtained by means of the above described shadow 
analysis. With this profile as initialization, a new surface profile is calculated by means of the 
shape-from-shading method. This process is iteratively repeated until the average change of the 
height profile between the two sequential iteration steps is smaller than a predetermined 
threshold value. 
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[0017] In the shape-from-shading methods known in the state of the art the surface profile is 
many times taken into consideration by the iterative minimization of an error function, which the 
deviation between the real and the from the surface profile reconstructed image are taken into 
consideration, as well as additional edge conditions placed upon the surface, such as for example 
flatness of the surface and integratability of the surface gradients. In particular manner this 
iterative minimization can be improved to the extent that, in the framework of the error function 
to be optimized in the iteration with the shape-from-shading method, an additive term is 
introduced. Therewith this additive term describes the deviation of the height difference 
determined in the previous iteration step in the light incident direction from the corresponding 
height differential determined by means of shadow analysis. For initialization of this 
advantageous iterative minimization there can be employed, in particularly preferred manner, the 
result obtained in the method described in the previous sentence. 

[0018] In the described simplest case of an evaluation of only two images (for example 
left/right illumination) structures cannot be sufficiently detected which, on the basis of their 
position or shape (here for example course of a rise from left to right), provide practically 
identical images in both images. These structures can however be identified simply by repeating 
the process with illumination from other sides in corresponding manner (here for example 
above/below illumination), so that finally all profile parts can be reconstructed by repeated 
application of the process with different incident angle. 

[0019] The inventive process is characterized by a very broad area of application. Surfaces 
of astronomical objects with great distances can be examined, as well as common products in the 
industrial manufacture, all the way to small structures under a microscope. Three examples of 
application of the surface profile determination and reconstruction illustrate this: 
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Reconstruction of Structures on the Moon Surface 



[0020] In Fig. 1 a part of the moon surface is reproduced, as it can be observed by a 
conventional telescope (here: 125 mm mirror telescope, 2600 mm focal point, CCD-camera). In 
the shown region the formation "Hesiodus" and "Wolf can be recognized (hereafter referred to 
as H or as the case may be W). A detailed section approximately between these formations is 
shown amplified or contrast enhanced and enlarged in Fig. 2.1 and 2.2. Therein the partial image 
2.1 of this area shows a light incidence from left and partial image 2.2 the area with light incident 
from the right, wherein the incidence of some light is respectively at a flat angle of 
approximately 4° to the surface of the moon. Here the sideways shadow fields of the rises are 
clearly to be recognized. Fig. 3 shows the contrast amplified extraction of the shadow fields by 
computer imaging of the quotients Ii/I 2 (or as the case may be I 2 /Ii) of the light intensity values 
of corresponding image points of both images. The therefrom determined BCC-analysis 
determined contours are entered as perimeter outlines in the respective partial image of Fig. 4. 
Fig. 5 shows in addition the determined perimeter outlines of the device or, as the case may be, 
crater flanks (lines about light areas), as well as the positions of the peaks of the rises or, as the 
case may be, crater edges as determined from the contours. 

[0021] Fig. 6 shows a perspective representation of a reconstructive profile of the analyzed 
moon region. 



Application Example 2 



Reconstruction of Structures on Work Pieces 



[0022] In the area of the industrial quality control the inventive process can be used for 
example for surface checking of components. Therein various structures can be examined: 
burrs, welded seams, convex and concave deformations, contraction cavities and inclusions, etc. 
This is currently frequently accomplished by manual "feeling" or by means of photogrammetrics 



{WP296867;!} 



-8- 



Attorney Docket 3926.236 



Patent Application 



with very high calibration expense with laboratory equipment. By the new process the 
calibration expense becomes unnecessary. Besides this, in the case of a suitable camera F-stop, 
objects even with a comparatively large distance from the camera can be measured. The image 
recorded by the camera can therein supplementally be compared with stored images, for example 
from CAD-data produced image information ("wire model"), and thus also slight deviations 
from the intended profile can be detected. 



[0023] In the two partial images from Fig. 7 a device is reproduced upon an otherwise 
planar work piece surface, wherein the illumination is incident from left to right, on the other 
hand from right to left, at a flat or shallow angle to the surface. Fig. 8 shows the height profile 
determined in accordance with the invention from these images of the device, which exhibits a 
height of up to approximately 5 mm. Two perspective views of the reconstructed burr are shown 
in Fig. 9. 

[0024] Further, significantly smaller structures can be thusly analyzed: Fig. 10 shows in 
two partial images (grazing or glancing illumination from different sides) of the surface segment 
of a further examined work piece (metal plate) with relatively small defects (two parallel running 
burrs). The burrs themselves are almost unrecognizable in the case of the perpendicular camera 
position, are however clearly visible by shadows of the burrs produced by side illumination 
(dashed lines). After evaluation of these images with the inventive process the profile of the 
burrs can be reconstructed as shown in Fig. 11. The height of the burr corresponds in this case to 
approximately 1.5 (or as the case may be 0.6 mm). 

Application Example 3 



Reconstruction of a Stamping with Integrated "Shape-from-shading" 

[0025] Fig. 13 shows a work piece of metal with a stamping (section) with illumination 
from different sides. In Fig. 14 the segments are reproduced, the arrows indicate the direction of 
incidence of light, the right partial images are the reconstructions obtained following the 
inventive process. Fig. 15 is the perspective representation of the results of the integrated shape- 
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from-shading, so that the reconstructed flanks and depressions have good correspondence with 
the original. The stamping depth corresponds in this example to approximately 0.4 mm. 

[0026] The inventive process can be modified in various manners. For example, the 
illumination of the object to be examined can occur using a single light source, which is pivoted 
about the corresponding axis of rotation (based on camera to object). With the same result the 
light source can be locationally fixed and the object can be rotated (same axis of rotation). 

[0027] With multiple light sources, pivoting or rotation mechanisms can be dispensed with. 
Therein the images can record sequentially with respectively one turned-on light source. In the 
case of use of light sources of different wave length (for example red and green) the images can 
also be recorded simultaneously and be separated by color images of the respective illuminated 
images. As an example, Fig. 12 shows a super-positioning of such color filter images. 

[0028] The term "light" is herein not limited to the visible spectrum, that is also IR or UV 
light sources can be employed with appropriate camera equipment. 

[0029] For image recording, all cameras with appropriate resolution can be used. Basically 
there applies herein: higher resolution brings about higher reconstruction precision. Today's 
CCD-cameras offer high resolution and provide digital images, which are available directly for 
evaluation using EDV. Therewith, using commercially available components (camera, laptop, 
stand, light source) a transportable measuring device can be constructed for carrying out the 
process. 
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ABSTRACT 



Disclosed is a A method for reconstructing the profile of structures on surfaces. 
According to said method, at least two images of the same area of the surface that is to be 
analyzed are evaluated, said images being recorded from a nearly vertical perspective, for 
example, and the surface being illuminated from different directions at a flat angle thereto. 
Elevations or depressions on the surface thus cast a distinct shadow in the recorded images, the 
position of which varies according to the incident light. Inclined areas can be identified by means 
of a brighter reflection. The vertical profile of a structure on the surface can be determined, and 
the shape of a fin can hence be reconstructed, for example, by analyzing shadow contours and 
outlines of bright areas. Even flat inclination changes can be determined by integrating the 
shape-from-shading process so as to evaluate brightness patterns such that a 3D reconstruction of 
the surface can be obtained, which matches the original well. 
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